An optical flow is a vector representing the motion of an object within an image by producing a velocity field of each point (pixel or area) within the image from temporally continuous images. One method for calculating an optical flow uses a gradient technique. By assuming that points within windows set in an image have optical flows with the same size as a constraint, the gradient technique derives a relational expression between a luminance gradient and an optical flow, and calculates the optical flow in a window unit from the relational expression.
However, when objects (for example, a plurality of different moving objects or moving objects and peripheral backgrounds thereof) having different optical flows exist within the same window, optical flow error becomes large. Therefore, a device disclosed in Patent Citation 1 calculates an optical flow of each area (each window) and optical flow error by the gradient technique using a plurality of images obtained from different times, and substituting an optical flow of a peripheral area as an optical flow of a target area when the optical flow error of the target area is greater than the optical flow error of the periphery area on the periphery of the target area.